Remote Assistance Method and System, and Electronic Device

ABSTRACT

A method, performed by a first terminal in a video call process between the first terminal and a second terminal, includes sending an image including a target device to a server, receiving device information corresponding to the target device sent by the server, virtualizing an operation interface based on the received device information, and displaying the operation interface on a current video call screen, where the operation interface is an operation interface of a control panel or a remote control of the target device, and displaying, on the current video call screen based on data from the second terminal, one or more operations performed on the operation interface by a user of the second terminal on the second terminal.

This application claims priority to Chinese Patent Application No.201911230080.9, filed with the China National Intellectual PropertyAdministration on Dec. 4, 2019 and entitled “REMOTE ASSISTANCE METHODAND SYSTEM, AND ELECTRONIC DEVICE”, which is incorporated herein byreference in its entirety.

TECHNICAL FIELD

The present invention relates to the field of augmented realitytechnologies, and in particular, to a remote assistance method andsystem, and an electronic device.

BACKGROUND

With rapid development of augmented reality (augmented reality, AR)technologies, the AR technologies are more widely used in fields such asentertainment, education, commerce, tourism, communication, and games,From fun applications such as MYOTee and AR games at the beginning to ARmeasurement, AR home, AR communication, and the like, AR applicationshave been more closely integrated with mobile terminals, and provideconsumers with better experience on the combination of virtual andreality.

In the prior art, a. user may use a mobile phone to connect, manage, andcontrol a household appliance, so that a state of the householdappliance can be controlled by one tap, and control and management canbe easily performed at home or outside. However, if the user does notknow how to use a function of the appliance, or if the user does notknow how to operate the appliance, or in other cases, the user can onlyobtain assistance generally through a call or a video call to learn howto operate, but if the operation is complicated and may not be easy tounderstand, or if the user requiring assistance has specificdifficulties such as vision and hearing impairment, this type ofguidance does not provide effective assistance.

SUMMARY

To overcome the foregoing problem, embodiments of this applicationprovide a remote assistance method and system, and an electronic device.

To achieve the foregoing objective, the following technical solutionsare used in the embodiments of this application:

According to a first aspect, this application provides a remoteassistance method. The method is performed by a first terminal in avideo call process between the first terminal and a second terminal, andincludes: sending an image including a target device to a server;receiving device information corresponding to the target device sent bythe server; virtualizing an operation interface based on the receiveddevice information, and displaying the operation interface on a currentvideo call screen, where the operation interface is an operationinterface of a control panel or a remote control of the target device;and displaying, on the current video call screen based on data from thesecond terminal, one or more operations performed on the operationinterface by a user of the second terminal on the second terminal.

In this embodiment of this application, a video stream including thetarget device is obtained, and the device information related to thedevice is obtained based on the video stream. Then, the virtual controlinterface is presented in a video of the first terminal by using an ARtechnology, so that the first terminal 20 obtains, by parsing based onoperation information sent by the second terminal, operation actions ofperforming an operation on the operation interface, and then the firstterminal restores these operation actions on the video screen based onthe operation information. By watching the restored operation actions, auser corresponding to the first terminal learns a method for operatingand controlling the target device, and can perform an operation directlyon the control panel on the first terminal or on a real remote controldevice. An assisted user can see, on a terminal of the user, a specificoperation performed on the real control panel or remote control. This isvery intuitive, and improves efficiency of communication between the twoparties.

In another possible implementation, the method further includes: bindingthe operation interface to the target device. This prevents the terminalfrom still using a virtual operation interface of a previous deviceafter the device in the video stream changes.

In another possible implementation, the displaying the operationinterface on a current video call screen includes: displaying theoperation interface on or near the target device on the current videocall screen, where the operation interface is the operation interface ofthe remote control of the target device; or overlaying or replacing thecontrol panel of the target device on the current video call screen withthe operation interface, where the operation interface is the operationinterface of the control panel of the target device.

In another possible implementation, the method further includes:receiving at least one first operation sent by the second terminal, andsending a control instruction to the target device, where the controlinstruction is used to control the target device.

In this embodiment of this application, a control party performs anoperation on a virtual interface in a video, and then sends a controlinstruction corresponding to the operation to the first terminal. Whenthe first terminal has a function of directly controlling the targetdevice, the first terminal directly sends the control instruction to thetarget device. In this way, an assisting party directly controls thetarget device, thereby helping the assisted party more effectively.

In another possible implementation, the method further includes:receiving at least one second operation performed by the user on theoperation interface; determining a visual effect corresponding to the atleast one second operation, and displaying the visual effect on thecurrent video call screen; and synchronizing the visual effect to thesecond terminal.

In this embodiment of this application, after the assisted partyperforms an operation on a virtual interface in a video, an effectcorresponding to the operation is sent to the terminal of the assistingparty, so that the assisting party knows whether the operation of theassisted party is correct.

In another possible implementation, the method further includes:determining, based on the received at least one second operation, a.control instruction corresponding to the at least one second operation,and sending the control instruction to the target device, where thecontrol instruction is used to control the target device.

According to a second aspect, an embodiment of this application providesa terminal device, including a transceiver, a processor, and a memory.The transceiver is configured to receive and send data. The memorystores one or more programs, the one or more programs include aninstruction, and when the instruction is executed by the processor, theelectronic device is enabled to perform the method according to thepossible implementations of the first aspect.

According to a third aspect, an embodiment of this application providesa remote assistance method. The method is performed by a second terminalin a video call process between the second terminal and a firstterminal. The method includes: receiving device information of a targetdevice, where the target device is a device displayed on a current videocall screen; virtualizing an operation interface based on the receiveddevice information, and displaying the operation interface on thecurrent video call screen, where the operation interface is an operationinterface of a control panel or a remote control of the target device;receiving at least one third operation performed by a user on theoperation interface on the current video call screen; and displaying avisual effect corresponding to one or more operations performed on theoperation interface, and synchronizing the visual effect to the firstterminal.

in this embodiment of this application, the video call screen includingthe virtual operation interface is presented on the second terminal. Auser corresponding to the second terminal performs an operation on thevirtual operation interface on the second terminal. The first terminalsynchronously presents the operation performed by the user on thevirtual operation interface, to assist a user corresponding to the firstterminal in controlling the target device.

In another possible implementation, the method further includes: bindingthe operation interface to the target device.

In another possible implementation, the displaying the operationinterface on the current video call screen includes: displaying theoperation interface on or around the target device on the current videocall screen, where the operation interface is a control interface of theremote control of the target device; or overlaying or replacing thecontrol panel of the target device on the current video call screen withthe operation interface, where the operation interface is a controlinterface of the control panel of the target device.

in another possible implementation, the method further includes:displaying, on the current video call screen based on data from thefirst terminal, one or more operations performed on the virtualoperation interface by a user of the first terminal on the firstterminal.

According to a fourth aspect, an embodiment of this application providesa terminal device, including a transceiver, a processor, and a memory.The transceiver is configured to receive and send data. The memorystores one or more programs, the one or more programs include aninstruction, and when the instruction is executed by the processor, theelectronic device is enabled to perform the method according to thepossible implementations of the third aspect.

According to a fifth aspect, an embodiment of this application providesa remote assistance system, including a target device that needs to becontrolled, a cloud server, a terminal device performing the methodaccording to the possible implementations of the first aspect, and aterminal device performing the method according to the possibleimplementations of the third aspect.

According to a sixth aspect, an embodiment of this application providesa readable storage medium, configured to store an instruction. When theinstruction is executed, the method according to the possibleimplementations of the first aspect and the method according to thepossible implementations of the third aspect are implemented.

According to a seventh aspect, an embodiment of this applicationprovides a computer program device including an instruction. When thecomputer program device runs on a terminal, the terminal is enabled toperform the method according to the possible implementations of thefirst aspect and the method according to the possible implementations ofthe third aspect.

BRIEF DESCRIPTION OF DRAWINGS

The following briefly describes the accompanying drawings required fordescribing the embodiments or the prior art.

FIG. 1 is a schematic structural diagram of a remote assistance systemaccording to an embodiment of this application;

FIG. 2 is a schematic diagram of a virtual operation interface displayedon a terminal according to an embodiment of this application;

FIG. 3 is a schematic diagram of a virtual operation interface displayedon a terminal according to an embodiment of this application;

FIG. 4 is a flowchart of a remote assistance method according to anembodiment of this application;

FIG. 5 is a flowchart of another remote assistance method according toan embodiment of this application;

FIG. 6A and FIG. 6B are a schematic flowchart of a communication processbetween a first terminal and a second terminal according to anembodiment of this application;

FIG. 7 is a schematic diagram of a working procedure between devices ina remote assistance system according to an embodiment of thisapplication; and

FIG. 8 is a schematic structural diagram of a terminal device accordingto an embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS

The following describes the technical solutions in the embodiments ofthis application with reference to the accompanying drawings in theembodiments of this application.

FIG. 1 is a schematic structural diagram of a remote assistance systemaccording to an embodiment of this application. As shown in FIG. 1 , thesystem includes a target device 10, a terminal 20, a terminal 30, and acloud server 40.

The target device 10 in this embodiment of this application is a device(target device) on which a control operation needs to be performed, andincludes but is not limited to an electronic device such as an airconditioner, a television, a mobile phone, or a computer, a mechanicaldevice, a vehicle, or another type of device, or is a remote control ofthe foregoing device. In this embodiment of this application, an airconditioner is used as an example to describe the technical solution.

The terminal 20 and the terminal 30 in this embodiment of thisapplication include but are not limited to electronic devices such asmobile phones, tablets, or notebooks.

In this embodiment of this application, in a video call process betweenthe terminal 20 and the terminal 30, the terminal 20 photographs thetarget device 10 by using a camera, to obtain an image of the targetdevice 10, and generate a video stream including the image of the targetdevice 10. The terminal 20 sends the obtained video stream including theimage of the target device 10 to the terminal 30, and also sends thevideo stream to the cloud server 40. The cloud server 40 identifies thetarget device 10 in the video stream in real time, and based on theidentified target device 10, matches a data model and controllableconfiguration file data that are corresponding to the target device 10,and then sends the data model and the controllable configuration tiledata that are corresponding to the target device 10 to the terminal 20and the terminal 30. The terminal 20 and the terminal 30 virtualize anoperation interface in a video based on the received data model. Theoperation interface is an image corresponding to a real remote controldevice or control panel that controls the target device 10, and ispresented at a position such as a position around or right above thetarget device 10 in the video. In addition, based on the receivedcontrollable configuration file, a button, a knob, and the like on thevirtualized operation interface are associated with correspondingcontrol instructions, so that when a user operates a control such as thebutton or the knob on the operation interface, the terminal 20 candetermine a control instruction corresponding to each operation.

In a specific example, as shown in FIG. 1 , after photographing an airconditioner, the terminal 20 sends a video including the air conditioneror an image of the air conditioner to the server, and the serveridentifies a brand and a model of the air conditioner through, forexample, artificial intelligence (artificial intelligence, AI), obtainsa two-dimensional or three-dimensional image model of a remote controlcorresponding to the air conditioner of the brand and the model, andobtains a controllable configuration file of an instructioncorresponding to each button on the remote control. For example, acontrollable configuration file corresponding to an “ON/OFF” button isthat an instruction for turning off or turning on the air conditioner isexecuted after the “ON/OFF” button is pressed. Then, the server sendsboth the remote control model and the controllable configuration file tothe terminal 20. After receiving the remote control model and thecontrollable configuration file, the terminal 20 virtualizes acorresponding remote control image, and displays the remote controlimage on a current video screen. The video screen with the virtualremote control image may be synchronized to the terminal 30 by using anaudio and video synchronization technology. Certainly, the server mayalternatively send the remote control model and the controllableconfiguration file to the terminal 30. However, the former is simplerand more economical during implementation. When the user operates avirtual button on the virtual remote control image, for example,operates the virtual button on the terminal 30, the terminal 30determines an instruction corresponding to the virtual button (differentoperations on a same button may correspond to different instructions,for example, a touch and hold operation and a short tap operationcorrespond to different instructions), sends information correspondingto the instruction, such as all operation position, an operation type,and a timestamp, to the peer terminal 20, and presents a visual effectcorresponding to the operation on the virtual remote control image onthe current video screen of the terminal 30. For example, the button isrecessed downward as the user presses, and is recovered as the userraises the hand. For example, after a touch button is pressed, there isa corresponding indicator, or LCD display content changes. The visualeffect may be included in the controllable configuration file. Theterminal determines a corresponding visual effect based on a currentuser operation and the controllable configuration file. The visualeffect is also synchronized to the terminal 20. In this way, the user ofthe terminal 20 can intuitively see how to control the remote control.Based on the received instruction, the terminal 20 may further send acontrol instruction for controlling the target device, to implementremote control on the target device.

j It should be noted that if the terminal 20 does not have a function ofcontrolling the target device 10 (for example, the terminal 20 does nothave an infrared emission function, or is not connected to the targetdevice), or if the user does not need to use the terminal 20 to directlycontrol the target device 10, or in other cases, the server 40 may notsend the controllable configuration file, or the sent controllableconfiguration file does not include a control instruction (aninstruction for directly controlling the target device). Based on anoperation that is displayed on the terminal 20 and that is performed bya remote assisting party on the remote control model, the user can learnan operation of controlling the target device 10, and can directlyperform an operation on the target device 10 or on a physical remotecontrol, to control the target device 10.

When the user performs an operation on the virtual operation interfaceon the terminal 20 or the terminal 30, the terminal 20 or the terminal30 records operation information in an operation process, for example,information such as a coordinate point, a timestamp, a quantity oftimes, and duration, determines a corresponding visual effect, presentsthe visual effect on the current video call screen, and synchronizes thevisual effect to the peer end of the video call. In a possibleembodiment, when user :A corresponding to the terminal 20 does not knowhow to operate the air conditioner, user B corresponding to the terminal30 assists user A in controlling the air conditioner, and user Ainforms, by using an audio and a video, user B that user A wants to“turn on the air conditioner and adjust a temperature to 21 degrees(assuming that a startup temperature is 17 degrees)”. User B performs anoperation on the virtual “ON/OFF” button of the operation interface ofthe virtual remote control displayed on the terminal 30, to turn on theair conditioner. In this case, the terminal 30 records operationinformation such as a coordinate point of a sliding operation, anoperation time, and a quantity of taps in the operation process. Then,user B performs an operation on a virtual temperature button “+F” of thevirtual remote control displayed on the terminal 30, to adjust thetemperature to “21 degrees” by tapping four times. In this case, theterminal 30 records operation information such as a coordinate point ofa sliding operation, an operation time, and a quantity of taps in theoperation process. In a user operation process, the terminal 20determines and presents visual effects of these operations based onreceived input. For example, a button of the virtual remote control ispressed, and a display window displays a corresponding icon, number, andthe like. These visual effects are displayed on a currently ongoingvideo call screen and synchronized to the peer end of the video call.The terminal 20 presents the visual effects of the user operations, toinform, in an intuitive manner, user A of how to perform an operation onthe terminal 20, so as to perform an operation on the air conditioner toturn on the air conditioner and adjust the temperature to 21 degrees.

FIG. 2 is a schematic diagram of a virtual operation interface displayedon a terminal according to an embodiment of this application.

In a possible embodiment, a terminal 20 and a terminal 30 are performinga video call by using a mobile phone, and a user of the terminal 20points a camera of the mobile phone to an air conditioner. In this way,an image of the air conditioner is displayed on a video call screen, asshown in FIG. 2 . The terminal 20 has a transmission function such asinfrared, Wi-Fi, Bluetooth, or data traffic, or has a function ofdirectly controlling a target device. After receiving a data model (anda controllable configuration file) corresponding to the air conditioner,the terminal 20 virtualizes a virtual control interface, and the controlinterface includes some virtual buttons such as a virtual “ON/'OFF”button, a virtual temperature button, and a virtual humidity button. Thevirtual control interface is presented near the air conditioner in avideo stream. After user B of the terminal 30 operates a virtual buttonon the virtual control interface, the terminal 30 presents visualeffects of these operations on the interface of the terminal 30,synchronizes the visual effects to the peer end of the video call,namely the terminal 20, and also sends obtained control informationcorresponding to the operation information to the terminal 20. User A ofthe terminal 20 can intuitively learn, by watching, how to perform anoperation on the control interface. After user A performs an operationon the virtual control interface of the terminal 20, the terminal 20sends a control instruction corresponding to each virtual button to theair conditioner in a transmission mode such as infrared, Wi-Fi,Bluetooth, or data traffic.

FIG. 3 is a schematic diagram of a virtual operation interface displayedon a terminal according to an embodiment of this application.

In a possible embodiment, as shown in FIG. 3 , after receiving a datamodel corresponding to an air conditioner, a terminal 20 constructs athree-dimensional structural diagram of a virtual air conditioner of thesame size as the air conditioner in a video stream, and then overlays orreplaces the air conditioner in the video stream with thethree-dimensional structural diagram of the virtual air conditioner. Thevirtual air conditioner is synchronized to a terminal 30 by using anaudio and video call synchronization technology. When user B performs anoperation on an operable button on the three-dimensional structuraldiagram of the virtual air conditioner on the terminal 30, the terminal30 synchronizes, to the terminal 20 in real time, a visual effectcorresponding to the operation on the three-dimensional structuraldiagram of the virtual air conditioner on a current video screen. Theterminal 20 presents the visual effect. By watching these restoredoperation actions, user A can learn a method for operating andcontrolling the air conditioner, and then directly perform an operationon a control panel of the air conditioner to control the airconditioner.

After receiving the operation performed by the user on the virtualoperation interface, the terminal needs to determine an operation objectbased on coordinates, and determine a display effect based oninformation such as a time, pressure, a direction, and a distance, and acontrollable configuration file. For example, if the user presses atouch button or a virtual button, a cursor or a shadow is displayed onthe button to indicate that the user selects the button, or an animationeffect such as a light circle or a special symbol is displayed on thebutton to indicate a corresponding operation such as touching andholding the button or tapping the button. There is a correspondenceamong operation information, an operation object, and a display effect,and the correspondence needs to be prestored locally or in the cloud.For example, the correspondence may be a controllable configurationfile, or may be obtained by the terminal from a network or anotherdevice after the terminal obtains a brand and a model of a targetdevice. In a remote assistance system provided in an embodiment of thisapplication, a terminal 20 obtains two-dimensional or three-dimensionaldata of a control interface, such as a control device or a controlpanel, corresponding to a target device 10, and presents a virtualcontrol interface corresponding to the data on a screen of a video callbetween the terminal 20 and a terminal 30. The virtual control interfaceis also synchronized to the terminal 30. The terminal 30 receives anoperation performed by an assisting party (user B) on the virtualcontrol interface presented on the terminal 30, and then sends theoperation information to the terminal 20. The terminal 20 restores theseoperation actions on the video screen based on the received operationinformation. Alternatively, based on a user operation, the terminal 30determines and presents the user operation, and synchronizes the useroperation to the peer end. In this way, by watching the operationaction, user A corresponding to the terminal 20 can learn a method foroperating and controlling the target device, and can perform anoperation directly on the control panel on the terminal 20 or on a realremote control device. Assisted user A can see, on the terminal of userA, a specific operation performed on the real control panel or remotecontrol. This is very intuitive as if the assisting party gives guidancein person on the scene, and improves efficiency of communication betweenthe two parties.

FIG. 4 is a flowchart of a remote assistance method. according to anembodiment of this application. As shown in FIG. 4 , an embodiment ofthis application provides a remote assistance method. The method isperformed by a terminal 20 corresponding to an assisted person. In avideo call process between the terminal 20 and a terminal 30,implementation steps of the method are as follows:

Step S401: Send an image including a target device to a server.

Before step S401, a communication connection is established between theterminal 20 and the terminal 30, to transmit data such as a video,device information, and operation information. In this embodiment ofthis application, a communication connection may be directly establishedbetween the terminal 20 and the terminal 30, for example, by using a D2Dcommunication technology. Alternatively, a connection may be establishedbetween the terminal 20 and the terminal 30 by using a video server. Forexample, in an existing WeChat video call process, the video serversends a video stream of the terminal 20 to the terminal 30 forpresentation. A specific implementation is not limited in thisapplication.

The communication connection between the terminal 20 and the terminal 30may be initiated by the terminal 20. When user A corresponding to theterminal 20 cannot perform an operation on a current target device 10,user B corresponding to the terminal 30 is specified to assist user A inperforming an operation on the target device 10. User A establishes avideo connection by initiating a request to the terminal 30.

After the communication connection is established between the terminal20 and the terminal 30, in the video call process, the terminal 20photographs the target device 10 by using a camera, to obtain an imageof the target device 10, generate a video stream including the image ofthe target device 10, and send the video stream to the terminal 30 inreal time. At the same time, the terminal 20 sends the video stream to acloud server 40.

Alternatively, the terminal 20 may send the cloud server 40 a picture, ashort video, or the like that includes the target device 10. In apossible embodiment, the terminal 20 sets a video stream to be obtained.For example, the terminal 20 captures pictures of N frames every severalseconds or minutes, and then sends the captured pictures to the cloudserver 40. The cloud server 40 identifies the tartlet device 10 in thepictures in real time. This reduces running load of the terminal 20 andreduces traffic, compared with the manner of sending the video stream tothe cloud server 40 in real time.

Step S403: Receive device: information corresponding to the targetdevice sent by the server.

Specifically, after the terminal 20 sends the video stream that includesthe target device 10, the cloud server 40 performs AI recognition toidentify information such as a brand and a model of the target device10, and then determines a device type of the target device 10. After thedevice type is determined, the device information corresponding to thedevice is matched. The device information includes a data model of thetarget device 10, and may further include controllable configurationfile data. The data model may be a real remote control used to controlthe target device 10 in FIG. 1 , a virtual operation interface used tocontrol the target device 10 in FIG. 2 , a control panel of the targetdevice 10 in FIG. 3 , or the like. The controllable configuration filemay be control instructions such as “turn on”, “turn off”, “turn up”“turn down”, and “channel”. Each control instruction in the controllableconfiguration file is associated with a button, a knob, or the like onan operation interface in the data model. In this way, when the useroperates a control such as the button or the knob on the operationinterface, the terminal 20 can determine a control instructioncorresponding to each operation. For example, a controllableconfiguration file corresponding to the “OFF” button is that aninstruction for turning off the target device 10 is executed after the“OFF” button is pressed.

Step S405: Virtualize an operation interface based on the receiveddevice information, and display the operation interface on a currentvideo call screen.

If the terminal 20 does not have a function of controlling the targetdevice 10 (for example, the terminal 20 does not have an infraredemission function, or is not connected to the target device), or if theuser does not need to use the terminal 20 to directly control the targetdevice 10, or in other cases, the user may select data content to herequested or received from the server, for example, by using a providedmenu option or voice prompt. The cloud server 40 may send only the datamodel corresponding to the target device 10. After receiving the datamodel, the terminal 20 presents, on a current video screen, an image ofa physical object such as a real remote control device, a real controlpanel on the tartlet device 10, or a control device corresponding to thetarget device 10.

In a first possible embodiment, as shown by the terminal 20 in FIG. 1(actually, interfaces displayed on the terminal 20 and the terminal 30are the same, for details, refer to the display interface of theterminal 30). after receiving a data model (and controllableconfiguration file data) of an air conditioner, the terminal 20 maychoose to virtualize, on the terminal 20, an air conditioner remotecontrol matching the air conditioner (a control interface displayed on avideo screen may be selected by the user or may be the default). Thevirtual air conditioner remote control is the same as a real airconditioner remote control, and includes buttons such as a virtual“ON/OFF” button used to control startup or shutdown of the airconditioner, and virtual buttons “−” and “+” used to control atemperature of the air conditioner. The terminal 20 synchronizes thedisplayed air conditioner remote control to the terminal 30, and user Bcorresponding to the terminal 30 can operate a button of the airconditioner remote control displayed on the video screen.

Even if the terminal 20 and the terminal 30 do not obtain a controllableconfiguration file, when the user operates the button of the airconditioner remote control on the video screen, the terminal stillpresents a visual effect corresponding to the operation, and then sendsthe visual effect to the peer terminal in real time, so that user A canview an operation effect and learn an operation for controlling thetarget device 10. In addition, if user A performs an operation on thevirtual air conditioner remote control of the terminal 20, user B canalso view the operation of user A, and therefore can provide guidanceand assistance for user A in real time.

In a second possible embodiment, as shown by the virtual operationinterface displayed on the terminal in FIG. 2 , the terminal 20 receivesa data model and a controllable configuration file of an airconditioner, and chooses to virtualize, on the terminal 20, a virtualoperation interface used to control the air conditioner. The virtualoperation interface includes buttons such as a virtual “ON/OFF” buttonused to control startup or shutdown of the air conditioner, and upwardand downward virtual buttons used to control a temperature of the airconditioner. The terminal 20 sends the displayed virtual operationinterface to the terminal 30, and user B corresponding to the terminal30 can operate a button on the virtual operation interface.

Because the terminal 20 receives the controllable configuration file ofthe air conditioner, if user A operates a button on the virtualoperation interface on the video screen, the terminal 20 sends, based onan instruction for controlling the air conditioner of a controllableconfiguration file corresponding to the button, the instruction to theair conditioner in a transmission mode such as infrared, Bluetooth, ordata traffic, to directly control the air conditioner.

In the foregoing two possible embodiments, the terminal 20 needs to bindthe received device information to the target device 10 in the videostream, so that the virtual operation interface displayed by theterminal 20 based on the device information corresponds to the targetdevice 10, to prevent the terminal 20 from still using a virtualoperation interface of a previous device after the target device 10 inthe video stream changes. Specifically, the terminal 20 continuously orintermittently sends the image of the target device 10 to the cloudserver 40. if the device image identified by the cloud server 40 doesnot change, the device information of the target device 10 sent to theterminal 20 is not to be changed. If the cloud server cannot identifythe image of the target device 10, the cloud server sends notificationinformation to the terminal 20, so that the terminal 20 stops displayingthe virtual operation interface in the video.

Step S407: Display, on the current video call screen, an operationperformed on the virtual operation interface by a user of a secondterminal.

After receiving the device information sent by the cloud server 40, theterminal 20 presents the virtual operation interface in the videostream. In addition, the device information is sent to the terminal 30(synchronized to the peer end by using an audio and videosynchronization algorithm or sent separately), so that the video callscreen including the virtual operation interface is synchronized to theterminal 30. This step is to ensure that content displayed on theterminal 20 and the terminal 30 is the same. In this way, the virtualoperation interface is also presented on the terminal 30 to assist userA in controlling the device.

The cloud server 40 may directly deliver the device information to theterminal 20 and the terminal 30. The terminal 20 does not need toforward the device information, thereby reducing workload of theterminal 20. However, the manner in which the terminal 20 synchronizesthe virtual operation interface is simpler during implementation.

After user B corresponding to the terminal 30 performs an operation onthe virtual operation interface, the terminal 30 records operationinformation such as a coordinate point of a sliding operation, anoperation time, a quantity of taps, and pressing duration in theoperation process, and sends the operation information to the terminal20. In addition, a visual effect corresponding to the operation ispresented on the virtual operation interface on the current video callscreen of the terminal 30. For example, a button is recessed downward asthe user presses, and is recovered as the user raises the hand. Thevisual effect is synchronized to the terminal 20, so that the user ofthe terminal 20 can intuitively see how to operate the remote control. Aspecific synchronization algorithm is the prior art. For example,synchronization is implemented by using an audio and videosynchronization algorithm or a timestamping manner. Details are notdescribed herein.

In this embodiment of this application, a plurality of methods may beused to control the target device 10. For example, in a possibleembodiment, the terminal 20 establishes a smart home interconnectionprotocol with the target device. After receiving the operationinformation sent by the terminal 30, the terminal 20 parses theoperation information, converts the operation information into aninstruction by using the smart home interconnection protocol, and sendsthe instruction to the air conditioner in a transmission mode such asinfrared, Wi-Fi, Bluetooth, or data traffic. In this way, user Acorresponding to the terminal 20 does not need to perform an operationagain. Alternatively, the terminal 30 may determine, based on theoperation of the user, a control instruction corresponding to theoperation of the user, and send the control instruction to the terminal20. Accordingly, the terminal 20 sends the control instruction to thetarget device by using., for example, an infrared instruction, aBluetooth instruction, a Wi-Fi instruction, or a point-to-pointinstruction, to control the target device.

In another possible embodiment, if the terminal 20 does not have aninfrared emission function, or is not connected to the target device 10,or in other cases, the terminal 30 presents, on the virtual operationinterface on the current video screen, a visual effect corresponding tothe operation, and sends the visual effect to the terminal 20 in realtime. By viewing the visual effect corresponding to the operation, theuser corresponding to the terminal 20 can learn a method for operatingand controlling the air conditioner, and then directly perform anoperation on a physical object such as a physical remote control, acontrol panel on the target device 10, or a control device correspondingto the target device 10.

It should be further noted that, after user A performs an operation onthe operation interface on the terminal 20, the terminal 20 presents, onthe virtual operation interface on the current video screen, a visualeffect corresponding to the operation, and sends the visual effect tothe terminal 30 in real time. By viewing the visual effect correspondingto the operation, user B corresponding to the terminal 30 determineswhether the operation performed by user A corresponding to the terminal20 is correct, and can provide guidance in a timely manner, for example,by using a voice and/or by performing an operation on the virtualoperation interface.

Similarly, after performing an operation on the operation interface onthe terminal 20, user A needs to send the operation information to theterminal 30. In a possible embodiment, user A draws a marked circle on ascreen of the terminal 20 for an area, and the terminal 20 needs torecord operation information such as a series of coordinate points, atimestamp, a quantity of times, and pressing duration that are on thescreen at that time, to present a corresponding visual effect. Thevisual effect is synchronized to the peer end. For example, a video sentby the terminal 20 is matched by using the timestamp, to ensure thatcontent seen by the two parties on the screen is the same.

This application provides the remote assistance method performed by theterminal 20. After a video stream including the target device 10 isobtained and related device information of the device is obtained basedon the video stream, a virtual control interface is presented in a videoon the terminal 20 by using an AR technology, and the virtual controlinterface is presented on video call screens of terminals of all partiesin communication by using an audio and video synchronization technology.A visual effect of an operation performed by a user of any terminal onthe virtual control interface is synchronized to all parties in thevideo call. In this way, by watching these operation actions, a userthat needs assistance, for example, user A corresponding to the terminal20, can learn a method for operating and controlling the target device,and can perform an operation directly on the control panel on theterminal 20 or on a real remote control device. Assisted user A can see,on the terminal of user A, a specific operation performed on the realcontrol panel or remote control. This is very intuitive as if theassisting party gives guidance in person on the scene, improvesefficiency of communication between the two parties, and reduces thedifficulty of remote assistance.

FIG. 5 is a flowchart of another remote assistance method according toan embodiment of this application. As shown in FIG. 5 , an embodiment ofthis application provides another remote assistance method. The methodis performed by a terminal 30 corresponding to an assisting person. In avideo call process between a terminal 20 and the terminal 30,implementation steps of the method are as follows:

Step S501: Receive device information of a target device.

Before step S501, a communication connection is established between theterminal 20 and the terminal 30. The communication connection betweenthe terminal 20 and the terminal 30 may be initiated by the terminal 30.When user B corresponding to the terminal 30 learns that user Acorresponding to the terminal 20 cannot perform an operation on adevice, user B establishes a video connection by initiating a request tothe terminal 20. Certainly, the communication connection between theterminal 20 and the terminal 30 may alternatively be initiated by theterminal 20.

In a possible embodiment, the device information obtained by theterminal 30 may not be sent by the terminal 20. After identifying atarget device 10 and obtaining device information corresponding to thedevice, a cloud server 40 directly delivers the device information tothe terminal 20 and the terminal 30. The terminal 20 does not need toforward the device information, thereby reducing workload of theterminal 20.

Step S503: Display a virtual operation interface on a current video callscreen.

The terminal 30 can virtualize an operation interface based on thereceived device information. If the device information received by theterminal 30 only includes a data model corresponding to the targetdevice 10, after receiving the data model, the terminal 30 presents thevirtual operation interface on a current video screen, and the operationinterface may correspond to an image of a physical object such as a realremote control device, a real control panel on the target device 10, ora control device corresponding to the target device 10.

In a first possible embodiment, as shown by the terminal 30 in FIG. 1 ,after receiving a data model of an air conditioner, the terminal 30virtualizes, on the terminal 30, an air conditioner remote controlmatching the air conditioner. The virtual air conditioner remote controlis the same as a real air conditioner remote control, and includesbuttons such as a virtual “ON/OFF” button used to control startup orshutdown of the air conditioner, and virtual buttons “−” and “+” used tocontrol a temperature of the air conditioner. The terminal 20 sends thedisplayed air conditioner remote control to the terminal 30, and user Bcorresponding to the terminal 30 can operate a button of the airconditioner remote control on the video screen.

Alternatively, a virtual operation interface displayed on the terminal20 may be synchronized to the terminal 30 by using an audio and videosynchronization technology.

When user B operates the button of the virtual air conditioner remotecontrol on the video screen, the terminal 30 presents a visual effectcorresponding to the operation, and sends the visual effect to theterminal 20 in real time, so that user A can view an operation effectand learn an operation for controlling the target device 10.

It should be noted that whether the air conditioner remote control oranother control device such as a control panel of the air conditioner isvirtualized on the terminal 30 needs to be determined based on selectionby user A corresponding to the terminal 20. If presentation selected byuser A is the air conditioner remote control, the air conditioner remotecontrol is also presented on the terminal 30. This ensures thatoperation interfaces of control device images presented on the terminal20 and the terminal 30 are the same. Therefore, in this implementation,when sending the device information to the terminal 30, the terminal 20selects a data model in the device information, so that only anoperation interface of a control device image displayed on the terminal20 can be presented on the terminal 30.

If the terminal 30 receives a controllable configuration file of the airconditioner, the terminal 30 generates a corresponding controlinstruction based on an operation of the user, and sends the controlinstruction to the terminal 20. The terminal 20 may send the controlinstruction to the air conditioner in a transmission mode such asinfrared, Wi-Fi, Bluetooth, or data traffic, to directly control the airconditioner.

In the foregoing two possible embodiments, the terminal 30 needs to bindthe received device information to a corresponding real device in avideo stream, so that the virtual operation interface displayed by theterminal 30 based on the device information corresponds to the device,to prevent the terminal 30 from still using a virtual operationinterface of a previous device after the device in the video streamchanges. Details are described above, and are not described hereinagain.

Step S505: Display, on the current video call screen, an operationperformed on the virtual operation interface by a user.

User B corresponding to the terminal 30 performs an operation on thevirtual operation interface, and the terminal 30 records operationinformation such as a coordinate point of a sliding operation, anoperation time, a quantity of taps, and pressing duration in theoperation process, and sends the operation information to the terminal20. In addition, a visual effect corresponding to the operation ispresented on the virtual operation interface on the current video screenof the terminal 30. For example, a button is recessed downward as theuser presses, and is recovered as the user raises the hand. The visualeffect is synchronized to the terminal 20, so that the user of theterminal 20 can intuitively and synchronously see an operation performedon the remote control.

An operation performed by user B on the virtual operation interface onthe terminal 30 may alternatively be sent to the terminal 20 in anothermanner or by using another algorithm, to ensure that content seen by thetwo parties on the screen is the same.

This application provides the remote assistance method performed by theterminal 30, The video call screen including the virtual operationinterface is presented on the terminal 30. User B corresponding to theterminal 30 performs an operation on the virtual operation interface onthe terminal 30. The terminal 20 synchronously presents the operationperformed by user B on the virtual operation interface, to assist user Acorresponding to the terminal 20 in controlling the target device 10.

FIG. 6A and FIG. 6B are a schematic flowchart of a communication processbetween a first terminal and a second terminal according to anembodiment of this application. As shown in FIG. 6A and FIG. 6B, thecommunication process between the first terminal and the second terminalis specifically as follows:

Step S601: Establish a communication connection between a terminal 20and a terminal 30.

The communication connection is established to transmit data such as avideo, device information, and operation information.

In a possible embodiment, the communication connection between theterminal 20 and the terminal 30 may be initiated by the terminal 20.When user A corresponding to the terminal 20 cannot perform an operationon a current target device 10, user B corresponding to the terminal 30is specified to assist user A in performing an operation on the targetdevice 10. User A establishes a video connection by initiating a requestto the terminal 30.

In a possible embodiment, the communication connection between theterminal 20 and the terminal 30 may be initiated by the terminal 30.When user B corresponding to the terminal 30 learns that user Acorresponding to the terminal 20 cannot perform an operation on adevice, user B establishes a video connection by initiating a request tothe terminal 20.

In this embodiment of this application, a communication connection maybe directly established between the terminal 20 and the terminal 30, forexample, by using a D2D communication technology. Alternatively, aconnection may be established between the terminal 20 and the terminal30 by using a video server. For example, in an existing WeChat videocall process, the video server sends a video stream of the terminal 20to the terminal 30 for presentation.

A specific implementation is not limited in this application.

Step S602: Start a camera of the terminal 20 to obtain a video streamincluding the target device 10.

After the communication connection is established between the terminal20 and the terminal 30, the terminal 20 starts the camera to photographthe target device 10, to obtain an image of the target device 10 andgenerate a video stream including the image of the target device 10.

Step S603: The terminal 20 sends the obtained video stream to theterminal 30 in real time.

In this embodiment of this application, after the connection isestablished between the terminal 20 and the terminal 30, the videostream of the terminal 20 may be sent to the terminal 30 in real timefor presentation, so that content displayed on the terminal 20 and theterminal 30 is the same.

Step S504: The terminal 20 sends the obtained video stream to a cloudserver 40 in real time.

Alternatively, the terminal 20 may send the cloud server 40 a picture, ashort video, or the like that includes the target device 10. In apossible embodiment, the terminal 20 sets a video stream to be obtained.For example, the terminal 20 captures pictures of N frames every severalseconds or minutes, and then sends the captured pictures to the cloudserver 40. The cloud server 40 identifies the target device 10 in thepictures in real time. This reduces running load of the terminal 20 andreduces traffic, compared with the manner of sending the video stream tothe cloud server 40 in real time.

Step S605: The cloud server 40 identifies the target device 10 in theimage based on the received image that includes the target device 10,and obtains, by matching, device information corresponding to the devicebased on the target device 10.

Specifically, after the terminal 20 sends the video stream that includesthe target device 10, the cloud server 40 performs AI recognition toidentify information such as a brand and a model of the target device10, and then determines a device type of the target device 10. After thedevice type is determined, the device information corresponding to thedevice is matched. The device information includes a data model of thetarget device 10, and may further include controllable configurationfile data. The data model may be a real remote control used to controlthe target device 10 in FIG. 1 , a virtual operation interface used tocontrol the target device 10 in FIG. 2 , a control panel of the targetdevice 10 in FIG. 3 , or the like. The controllable configuration filemay be control instructions such as “turn on”, “turn off”, “turn up”,“turn down”, and “channel”. Each control instruction in the controllableconfiguration file is associated with a button, a knob, or the like onan operation interface in the data model. In this way, when the useroperates a control such as the button or the knob on the operationinterface, the terminal 20 can determine a control instructioncorresponding to each operation. For example, a controllableconfiguration file corresponding to the “OFF” button is that aninstruction for turning off the target device 10 is executed after the“OFF” button is pressed.

Step S606: The cloud server 40 sends the device information to theterminal 20,

If the terminal 20 does not have a function of controlling the targetdevice 10 (for example, the terminal 20 does not have an infraredemission function, or is not connected to the target device), or if theuser does not need to use the terminal 20 to directly control the targetdevice 10, or in other cases, the cloud server 40 may send only the datamodel corresponding to the target device 10. If the terminal 20 has atransmission function such as infrared, Wi-Fi, Bluetooth, or datatraffic, or has a function of directly controlling a target device, thecloud server 40 sends the data model and the controllable configurationfile that are corresponding to the target device 10.

Step S607: The terminal 20 virtualizes an operation interface in thevideo stream based on the device information.

After receiving the data model, the terminal 20 presents, on a currentvideo screen, an. image of a physical object such as a real remotecontrol device, a real control panel on the target device 10, or acontrol device corresponding to the target device 10. In a firstpossible embodiment, as shown by the terminal 20 in FIG. 1 (actually,interfaces displayed on the terminal 20 and the terminal 30 are thesame; for details, refer to the display interface of the terminal 30).after receiving a data model of an air conditioner, the terminal 20chooses to virtualize, on the terminal 20, an air conditioner remotecontrol matching the air conditioner (a control interface displayed on avideo screen may be selected by the user or may be the default). Thevirtual air conditioner remote control is the same as a real airconditioner remote control, and includes buttons such as a virtual“ON/OFF” button used to control startup or shutdown of the airconditioner, and virtual buttons “−” and “+” used to control atemperature of the air conditioner. The terminal 20 sends the displayedair conditioner remote control to the terminal 30, and user Bcorresponding to the terminal 30 can operate a button of the airconditioner remote control on the video screen.

Even if the terminal 20 and the terminal 30 do not have a controllableconfiguration file, when the user operates the button of the airconditioner remote control on the video screen, the terminal stillpresents a visual effect corresponding to the operation, and then sendsthe visual effect to the peer terminal in real time, so that user A canview an operation effect and learn an operation for controlling thetarget device 10. In addition, user B can also view an operation of userA, and therefore can provide guidance and assistance for user A in realtime.

After receiving the data model and the controllable configuration file,the terminal 20 may not only present, on a current video screen, animage of a physical object such as a real remote control device, a realcontrol panel on the target device 10, or a real control devicecorresponding to the target device 10, but also present the virtualoperation interface shown in FIG. 2 , In a second possible embodiment,as shown by the virtual operation interface displayed on the terminal inFIG. 2 , the terminal 20 receives a data model and a controllableconfiguration file of an air conditioner, and chooses to virtualize, onthe terminal 20, a virtual operation interface used to control the airconditioner. The virtual operation interface includes buttons such as avirtual “ON/OFF” button used to control startup or shutdown of the airconditioner, and upward and downward virtual buttons used to control atemperature of the air conditioner. The terminal 20 sends the displayedvirtual operation interface to the terminal 30, and user B correspondingto the terminal 30 can operate a button on the virtual operationinterface.

Because the terminal 20 receives the controllable configuration file ofthe air conditioner, if user A operates a button on the virtualoperation interface on the video screen, the terminal 20 sends, based onan instruction for controlling the air conditioner of a controllableconfiguration file corresponding to the button, the instruction to theair conditioner in a transmission mode such as infrared, Wi-Fi,Bluetooth, or data traffic, to directly control the air conditioner.

In the foregoing two possible embodiments, the terminal 20 needs to bindthe received device information to the target device 10 in the videostream, so that the virtual operation interface displayed by theterminal 20 based on the device information corresponds to the targetdevice 10, to prevent the terminal 20 from still using a virtualoperation interface of a previous device after the target device 10 inthe video stream charges. Specifically, the terminal 20 continuously orintermittently sends the image of the target device 10 to the cloudserver 40. If the device image identified by the cloud server 40 doesnot change, the device information of the target device 10 sent to theterminal 20 is not to be changed. If the cloud server cannot identifythe image of the target device 10, the cloud server sends notificationinformation to the terminal 20, so that the terminal 20 stops displayingthe virtual operation interface in the video.

Step S608: The terminal 20 sends the device information to the terminal30.

In a possible embodiment, the device information obtained by theterminal 30 may not be sent by the terminal 20. After identifying thedevice and obtaining the device information corresponding to the device,the cloud server 40 directly delivers the device information to theterminal 20 and the terminal 30. The terminal 20 does not need toforward the device information, thereby reducing workload of theterminal 20.

Step S609: The terminal 30 virtualizes an operation interface in thevideo stream based on the device information.

If the device information received by the terminal 30 only includes thedata model corresponding to the target device 10, after receiving thedata model, the terminal 30 presents, on a current video screen, animage of a physical object such as a real remote control device, a realcontrol panel on the target device 10, or a control device correspondingto the target device 10.

in a first possible embodiment, as shown by the terminal 30 in FIG. 1 ,after receiving a data model of an air conditioner, the terminal 30virtualizes, on the terminal 30 based on selection by the terminal 20,an air conditioner remote control matching the air conditioner. Thevirtual air conditioner remote control is the same as a real airconditioner remote control, and includes buttons such as a virtual“ON/OFF” button used to control startup or shutdown of the airconditioner, and virtual buttons “−” and “+” used to control atemperature of the air conditioner. The terminal 20 sends the displayedair conditioner remote control to the terminal 30, and user Bcorresponding to the terminal 30 can operate a button of the airconditioner remote control on the video screen.

When user B operates the button of the air conditioner remote control onthe video screen, the terminal 30 presents a visual effect correspondingto the operation, and sends the visual effect to the terminal 20 in realtime, so that user A can view an operation effect and learn an operationfor controlling the target device 10.

It should be noted that Whether the air conditioner remote control oranother control device such as a control panel of the air conditioner isvirtualized on the terminal 30 needs to be determined based on selectionby user A corresponding to the terminal 20. If presentation selected byuser A is the air conditioner remote control, the air conditioner remotecontrol is also presented on the terminal 30. This ensures thatoperation interfaces of control device images presented on the terminal20 and the terminal 30 are the same. Therefore, in this implementation,when sending the device information to the terminal 30, the terminal 20selects a data model in the device information, so that only anoperation interface of a control device image displayed on the terminal20 can be presented on the terminal 30.

If the device information received by the terminal 30 includes the datamodel and the controllable configuration file that are corresponding tothe target device 10, after receiving the data model and thecontrollable configuration file, the terminal 30 may not only present,on a current video screen, an image of a physical object such as a realremote control device, a real control panel on the target device 10, ora real control device corresponding to the target device 10, but alsopresent the virtual operation interface shown in FIG. 2 .

In a second possible embodiment, as shown by the virtual operationinterface displayed on the terminal in FIG. 2 , after receiving a datamodel and a controllable configuration file of an air conditioner, theterminal 30 virtualizes, on the terminal 30 based on selection by theterminal 20, a virtual operation interface used to control the airconditioner. The virtual operation interface includes buttons such as avirtual “ON/OFF” button used to control startup or shutdown of the airconditioner, and upward and downward virtual buttons used to control atemperature of the air conditioner. The terminal 20 sends the displayedvirtual operation interface to the terminal 30, and user B correspondingto the terminal 30 can operate a button on the virtual operationinterface.

Because the terminal 30 receives the controllable configuration file ofthe air conditioner, if user B operates a button on the virtualoperation interface on the video screen, after generated operationinformation is sent to the terminal 20, the terminal 20 sends, based onan instruction for controlling the air conditioner of a controllableconfiguration file corresponding to the button, the instruction to theair conditioner in a transmission mode such as infrared, Wi-Fi,Bluetooth, or data traffic, to directly control the air conditioner.

In the foregoing two possible embodiments, the terminal 30 needs to bindthe received device information to a real device in a video stream, sothat the virtual operation interface displayed by the terminal 30 basedon the device information corresponds to the device, to prevent theterminal 30 from still using a virtual operation interface of a previousdevice after the device in the video stream changes.

Step S610: The terminal 30 records operation information of an operationperformed by the user on the virtual operation interface.

User B corresponding to the terminal 30 performs an operation on thevirtual operation interface, and the terminal 30 records operationinformation such as a coordinate point of a sliding operation, anoperation time, a quantity of taps, and pressing duration in theoperation process. After determining that user B completes the operationin the control process, the terminal 30 sends the recorded operationinformation to the terminal 20. In addition, a visual effectcorresponding to the operation is presented on the virtual operationinterface on the current video screen of the terminal 30. For example, abutton is recessed downward as the user presses, and is recovered as theuser raises the hand. The visual effect is also synchronized to theterminal 20, so that the user of the terminal 20 can intuitively see howto operate the remote control.

It should be further noted that, after performing an operation on thevideo stream on the terminal 30, user B needs to send the operationinformation to the terminal 20, to ensure that content seen by the twoparties on the screen is the same.

Step S611: The terminal 30 sends operation information and/or a video tothe terminal 20.

Step S612: The terminal 20 parses the operation information, andconverts a parsing result into an operation action of performing anoperation on the operation interface.

After receiving and parsing the operation information, the terminal 20informs, in a manner such as video demonstration, user A of how toperform an operation on the terminal 20, to control the target device10, in addition, a visual effect corresponding to the operation ispresented on the virtual operation interface on the current video screenof the terminal 30. For example, a button is recessed downward as theuser presses, and is recovered as the user raises the hand. The visualeffect is also synchronized to the terminal 20, so that the user of theterminal 20 can intuitively see how to operate the remote control.

In this embodiment of this application, a plurality of methods may beused to control the target device 10. For example, in a possibleembodiment, the terminal 20 establishes a smart home interconnectionprotocol with the target device. After receiving the operationinformation sent by the terminal 30, the terminal 20 parses theoperation information, converts the operation information into aninstruction by using the smart home interconnection protocol, and sendsthe instruction to the air conditioner in a transmission mode such asinfrared, Wi-Fi, Bluetooth, or data traffic. In this way, user Acorresponding to the terminal 20 does not need to perform an operationagain.

In another possible embodiment, if the terminal 20 does not have aninfrared emission function, or is not connected to the target device 10,or in other cases, the terminal 30 presents, on the virtual operationinterface on the current video screen, a visual effect corresponding tothe operation, and sends the visual effect to the terminal 20 in realtime. By viewing the visual effect corresponding to the operation, theuser corresponding to the terminal 20 can learn a method for operatingand controlling the air conditioner, and then directly perform anoperation on a physical object such as a physical remote control, acontrol panel on the target device 10, or a control device correspondingto the target device 10.

Step S613: The terminal 20 sends operation information and/or a video tothe terminal 30.

Step S614: The terminal 30 presents the operation interface on theterminal 20.

After user A performs an operation on the video stream on the terminal20, the terminal 20 presents, on the virtual operation interface on thecurrent video screen, a visual effect corresponding to the operation,and sends the visual effect to the terminal 30 in real time. By viewingthe visual effect corresponding to the operation, user B correspondingto the terminal 30 determines whether the operation performed by user Acorresponding to the terminal 20 is correct.

Similarly, after performing an operation on the operation interface onthe terminal 20, user A needs to send the operation information to theterminal 30. In a possible embodiment, user A draws a marked circle on ascreen of the terminal 20 for an area, and the terminal 20 needs torecord operation information such as a. series of coordinate points, atimestamp, and a. quantity of times that are on the screen at that time.Then, the operation information is sent to the terminal 30 by using adedicated communications channel, and a video sent by the terminal 20 ismatched by using the timestamp, to ensure that content seen by the twoparties on the screen is the same.

A remote assistance system provided in an embodiment of this applicationis configured to implement the foregoing method embodiments.Specifically, after obtaining a video stream including a target device10, a terminal 20 sends the video stream to a terminal 30 and a server40. The server 40 obtains two-dimensional or three-dimensional data of areal control interface, such as a control device or a control panel,corresponding to the target device 10, and then sends thetwo-dimensional or three-dimensional data to the terminal 20 and theterminal 30. A virtual control interface is presented in videos on theterminal 20 and the terminal 30 by using an AR technology. The terminal30 receives an operation performed by an assisting party (user B) on thevirtual control interface, and then sends the operation information tothe terminal 20. The terminal 20 restores these operation actions on avideo screen based on the received operation information. By watchingthe restored operation actions, user A corresponding to the terminal 20can learn a method for operating and controlling the target device, andcan perform an operation directly on the control panel on the terminal20 or on a real remote control device. Assisted user A can see, on theterminal of user A, a specific operation performed on the real controlpanel or remote control. This is very intuitive as if the assistingparty gives guidance in person on the scene, and improves efficiency ofcommunication between the two parties.

For the technical solution in the foregoing “first possible embodiment”,this application lists an embodiment to facilitate understanding: It isassumed that an elderly person at home is user A corresponding to theterminal 20, and a son in an office is user B corresponding to theterminal 30. An “AR Remote Assistance” application is installed onsmartphones of the two people. When elderly person A needs to turn on asleep mode of an air conditioner but cannot find a remote control ordoes not know how to operate the remote control, elderly person A opensthe “AR Remote Assistance” application on the mobile phone to seek helpfrom son B. Elderly person A points (a camera of) the mobile phone atthe air conditioner. An image of the air conditioner is displayed on themobile phone screen of son B, and a remote control of the airconditioner is displayed on the screen. Son B operates a correspondingbutton on the remote control interface, and an instruction is returnedto the mobile phone of elderly person A. When elderly person A watchesthe son operating the remote control on the mobile phone of elderlyperson A, the mobile phone of elderly person A turns on the sleep modeof the air conditioner by using a smart home interconnection protocolafter receiving the air conditioner control instruction.

For the technical solution in the foregoing “second possibleembodiment”, this application lists an embodiment to facilitateunderstanding: It is assumed that an elderly person at home is user Acorresponding to the terminal 20, and a son in an office is user Bcorresponding to the terminal 30. An “AR Remote Assistance” applicationis installed on smartphones of the two people. Elderly person A needs toturn on a sleep mode of an air conditioner, a remote control of the airconditioner is in the hands of elderly person A, and elderly person Aopens the “AR Remote Assistance” application on the mobile phone to seekhelp from son B. Elderly person A points the mobile phone at the remotecontrol. An image of the remote control is displayed on the mobile phonescreen of son B. Son B operates a corresponding button on the remotecontrol interface, and an instruction is returned to the mobile phone ofelderly person A. When elderly person A watches the son operating theremote control on the mobile phone of elderly person A, the mobile phoneof elderly person A turns on the sleep mode of the air conditioner byusing the smart home interconnection protocol after receiving the airconditioner control instruction.

An embodiment of this application further provides a remote controlsystem, configured to implement the foregoing method embodiments:Specifically, the system includes a target device 10 that needs to becontrolled, a cloud server 40, a remote control apparatus 70 performedby a terminal 20, and a remote control apparatus 80 performed by aterminal 30. By using an AR technology, the system can superimpose avirtual operation interface on a video stream obtained by the terminal20. This facilitates communication between an assisting party and anassisted party, and makes it easier for the assisting party to assistthe assisted party in operating a device.

FIG. 7 is a schematic diagram of a working procedure between devices ina remote assistance system according to an embodiment of thisapplication. As shown in FIG. 7 , a process of implementing remoteassistance by the system mainly includes a communication establishmentphase, an AI recognition phase, an AR recognition phase, and a smarthome interconnection phase. Party A represents a terminal 20, party Brepresents a terminal 30, a cloud represents a cloud server 40. and anobject represents a target device 10.

Communication Establishment Phase:

Step 1: Establish a communication connection between party A and partyB.

The communication connection is established to transmit data such as avideo, device information, and operation information.

In a possible embodiment, the communication connection between theterminal 20 and the terminal 30 may be initiated by the terminal 20.When user A corresponding to the terminal 20 cannot perform an operationon a current target device 10, user B corresponding to the terminal 30is specified to assist user A in performing an operation on the targetdevice 10. User A establishes a video connection by initiating a requestto the terminal 30.

In a possible embodiment, the communication connection between theterminal 20 and the terminal 30 may be initiated by the terminal 30.When user B corresponding to the terminal 30 learns that user Acorresponding to the terminal 20 cannot perform an operation on adevice, user B establishes a video connection by initiating a request tothe terminal 20.

In this embodiment of this application, a communication connection maybe directly established between the terminal 20 and the terminal 30, forexample, by using a D2D communication technology. Alternatively, aconnection may be established between the terminal 20 and the terminal30 by using a video server. For example, in an existing WeChat videocall process, the video server sends a video stream of the terminal 20to the terminal 30 for presentation. A specific implementation is notlimited in this application.

Step 2: Party A uses a camera to obtain an object that needs to becontrolled.

After the communication connection is established between the terminal20 and the terminal 30, the terminal 20 starts the camera to photographthe target device 10, to obtain a two-dimensional or three-dimensionalstructural diagram of the target device 10 and generate a video streamincluding the two-dimensional or three-dimensional structural diagram ofthe target device 10.

Step 3: In a video call process between party A and party B, party Bobtains, in real time, video content photographed by party A.

In this embodiment of this application, after the connection isestablished between the terminal 20 and the terminal 30, the videostream of the terminal 20 may be sent to the terminal 30 in real timefor presentation, so that content displayed on the terminal 0 and theterminal 30 is the same.

AI Recognition Phase:

Step 1: The cloud identifies an object in the video in real time.

Alternatively, the terminal 20 may send the cloud server 40 a picture, ashort video, or the like that includes the target device 10. in apossible embodiment, the terminal 20 sets a video stream to be obtained.For example, the terminal 20 captures pictures of N frames every severalseconds or minutes, and then sends the captured pictures to the cloudserver 40. The cloud server 40 identifies the target device 10 in thepictures in real time. This reduces running load of the terminal 20 andreduces traffic, compared with the manner of sending the video stream tothe cloud server 40 in real time.

Step 2: Match operation information corresponding to the object afterinformation about the object is identified, where the operationinformation includes information such as turn on, turn off, turn up,turn down, or a channel.

After the terminal 20 sends the video stream that includes the targetdevice 10, the cloud server 40 performs AI recognition to identifyinformation such as a brand and a model of the target device 10, andthen determines a device type of the target device 10. After the devicetype is determined, the device information corresponding to the deviceis matched. The device information includes a data model of the targetdevice 10, and may further include controllable configuration file data.The data model may be a real remote control used to control the targetdevice 10 in FIG. 1 , a virtual operation interface used to control thetarget device 10 in FIG. 2 , a control panel of the target device 10 inFIG. 3 . or the like. The controllable configuration file may be controlinstructions such as “turn on”, “turn off”, “turn up”, “turn down”, and“channel”. Each control instruction in the controllable configurationfile is associated with a button, a knob, or the like on an operationinterface in the data model. In this way, when the user operates acontrol such as the button or the knob on the operation interface, theterminal 20 can determine a control instruction corresponding to eachoperation. For example, a controllable configuration file correspondingto the “OFF” button is that an instruction for turning off the targetdevice 10 is executed after the “OFF” button is pressed.

AR Recognition Phase:

Step 1: Party A obtains the information about the object from the cloud,implements a virtual operation panel, and binds the operation panel tothe object in the video.

In one case, if the terminal 20 does not have a function of controllingthe target device 10 (for example, the terminal 20 does not have aninfrared emission function, or is not connected to the target device),or if the user does not need to use the terminal 20 to directly controlthe target device 10, or in other cases, the cloud server 40 may sendonly the data model corresponding to the target device 10. Afterreceiving the data model, the terminal 20 presents, on a current videoscreen, an image of a physical object such as a real remote controldevice, a real control panel on the target device 10, or a controldevice corresponding to the target device 10. In a first possibleembodiment, as shown by the terminal 20 in FIG. 1 (actually, interfacesdisplayed on the terminal 20 and the terminal 30 are the same; fordetails, refer to the display interface of the terminal 30), afterreceiving a data model of an air conditioner, the terminal 20 chooses tovirtualize, on the terminal 20, an air conditioner remote controlmatching the air conditioner (a control interface displayed on a videoscreen may be selected by the user or ma be the default). The virtualair conditioner remote control is the same as a real air conditionerremote control, and includes buttons such as a virtual “ON/OFF” buttonused to control startup or shutdown of the air conditioner, and virtualbuttons “−” and “+” used to control a temperature of the airconditioner. The terminal 20 sends the displayed air conditioner remotecontrol to the terminal 30, and user B corresponding to the terminal 30can operate a button of the air conditioner remote control on the videoscreen.

Even if the terminal 20 and the terminal 30 do not have a controllableconfiguration file, when the user operates the button of the airconditioner remote control on the video screen, the terminal stillpresents a visual effect corresponding to the operation, and then sendsthe visual effect to the peer terminal in real time, so that user A canview an operation effect and learn an operation for controlling thetarget device 10. In addition, user B can also view an operation of userA, and therefore can provide guidance and assistance for user A in realtime.

In another case, if the terminal 20 has a transmission function such asinfrared, Wi-Fi, Bluetooth, or data traffic, or has a function ofdirectly controlling a target device, the cloud server 40 sends the datamodel and the controllable configuration file that are corresponding tothe target device 10. her receiving the data model and the controllableconfiguration file, the terminal 20 may not only present, on a currentvideo screen, an image of a physical object such as a real remotecontrol device, a real control panel on the target device 10, or a realcontrol device corresponding to the target device 10, but also presentthe virtual operation interface shown in FIG. 2 . In a second possibleembodiment, as shown by the virtual operation interface displayed on theterminal in FIG. 2 , the terminal 20 receives a data model and acontrollable configuration file of an air conditioner, and chooses tovirtualize, on the terminal 20, a virtual operation interface used tocontrol the air conditioner. The virtual operation interface includesbuttons such as a virtual “ON/OFF” button used to control startup orshutdown of the air conditioner, and upward and downward virtual buttonsused to control a temperature of the air conditioner. The terminal 20sends the displayed virtual operation interface to the terminal 30, anduser B corresponding to the terminal 30 can operate a button on thevirtual operation interface.

Because the terminal 20 receives the controllable configuration file ofthe air conditioner, if user A operates a button on the virtualoperation interface on the video screen, the terminal 20 sends, based onan instruction for controlling the air conditioner of a controllableconfiguration file corresponding to the button, the instruction to theair conditioner in a transmission mode such as infrared, Wi-Fi,Bluetooth, or data traffic, to directly control the air conditioner.

Step 2: Party B obtains the object information sent by party A, and alsovirtualizes an operation panel in the video, to obtain the same contentas that displayed on a display interface of party A.

The device information obtained by the terminal 30 may not be sent bythe terminal 20. After identifying the device and obtaining the deviceinformation corresponding to the device, the cloud server 40 directlydelivers the device information to the terminal 20 and the terminal 30.The terminal 20 does not need to forward the device information, therebyreducing workload of the terminal 20.

In one case, after receiving a data model of an air conditioner, theterminal 30 virtualizes, on the terminal 30 based on selection by theterminal 20, an air conditioner remote control matching the airconditioner. The virtual air conditioner remote control is the same as areal air conditioner remote control, and includes buttons such as avirtual “ON/OFF” button used to control startup or shutdown of the airconditioner, and virtual buttons “−” and “+” used to control atemperature of the air conditioner. The terminal 20 sends the displayedair conditioner remote control to the terminal 30, and user Bcorresponding to the terminal 30 can operate a button of the airconditioner remote control on the video screen.

In this case, when user B operates the button of the air conditionerremote control on the video screen, the terminal 30 still presents avisual effect corresponding to the operation, and then sends the visualeffect to the terminal in real time, so that user A can view anoperation effect and learn an operation for controlling the targetdevice 10.

In another case, after receiving a data model and a controllableconfiguration file of an air conditioner, the terminal 30 virtualizes,on the terminal 30 based on selection by the terminal 20, a virtualoperation interface; used to control the air conditioner. The virtualoperation interface includes buttons such as a virtual “ON/OFF” buttonused to control startup or shutdown of the air conditioner, and upwardand downward virtual buttons used to control a temperature of the airconditioner. The terminal 20 sends the displayed virtual operationinterface to the terminal 30, and user B corresponding to the terminal30 can operate a button on the virtual operation interface.

Because the terminal 30 receives the controllable configuration file ofthe air conditioner, if user B operates a button on the virtualoperation interface on the video screen, after generated operationinformation is sent to the terminal 20, the terminal 20 sends, based onan instruction for controlling the air conditioner of a controllableconfiguration file corresponding to the button, the instruction to theair conditioner in a transmission mode such as infrared, Wi-Fi,Bluetooth, or data traffic, to directly control the air conditioner.

Step 3: Party B performs a remote operation on the virtual operationpanel, and records operation information such as coordinate information,a timestamp, and a quantity of taps.

User B corresponding to the terminal 30 performs an operation on thevirtual operation interface, and the terminal 30 records operationinformation such as a coordinate point of a sliding operation, anoperation time, a quantity of taps, and pressing duration in theoperation process. After determining that user B completes the operationin the control process, the terminal 30 sends the recorded operationinformation to the terminal 20. In addition, a visual effectcorresponding to the operation is presented on the virtual operationinterface on the current video screen of the terminal 30. For example, abutton is recessed downward as the user presses, and is recovered as theuser raises the hand. The visual effect is also synchronized to theterminal 20, so that the user of the terminal 20 can intuitively see howto operate the remote control.

Step 4: After obtaining the operation information of party B, party Aobtains, by parsing based on the operation information, a specificbutton operation such as turn on, turn off, turn up, turn down, or achannel.

After receiving and parsing the operation information sent by theterminal, the terminal 20 informs, in a manner such as videodemonstration, user A of how to perform an operation on the terminal 20,to control the target device 10. In addition, a visual effectcorresponding to the operation is presented on the virtual operationinterface on the current video screen of the terminal 30. For example, abutton is recessed downward as the user presses, and is recovered as theuser raises the hand. The visual effect is also synchronized to theterminal 20, so that the user of the terminal 20 can intuitively see howto operate the remote control.

Smart Home Interconnection Phase:

Step 1: Party A controls the corresponding object based on the buttonoperation.

In one case, the terminal 20 establishes a smart home interconnectionprotocol with the target device. After receiving the operationinformation sent by the terminal 30, the terminal 20 parses theoperation information, converts the operation information into aninstruction by using the smart home interconnection protocol, and sendsthe instruction to the air conditioner in a transmission mode such asinfrared, Wi-Fi, Bluetooth, or data traffic. In this way, user Acorresponding to the terminal 20 does not need to perform an operationagain.

In another case, if the terminal 20 does not have an infrared emissionfunction, or is not connected to the target device 10, or in othercases, the terminal 30 presents, on the virtual operation interface onthe current video screen, a visual effect corresponding to theoperation, and sends the visual effect to the terminal 20 in real time.By viewing the visual effect corresponding to the operation, the usercorresponding to the terminal 20 can learn a method for operating andcontrolling the air conditioner, and then directly perform an operationon a physical object such as a physical remote control, a control panelon the target device 10, or a control device corresponding to the targetdevice 10.

FIG. 8 is a schematic structural diagram of a terminal device accordingto an embodiment of the present invention. The terminal device isconfigured to implement the foregoing method embodiments. FIG. 8 shows aterminal target device 800. The terminal target device 800 may be theterminal 20 or the terminal 30, and includes a sensor 801, a display802, a processor 803, a memory 804, a communications interface 805, anda bus 806. The processor 803, the memory 804, and the communicationsinterface 805 in the terminal device can establish a communicationconnection by using the bus 806.

The sensor 801 is configured to obtain a video stream including a deviceand device information corresponding to the device. The sensor 801 mayinclude a camera or the like.

The display 802 is configured to display processed data, such as a videoand a virtual operation interface.

The processor 803 may be a central processing unit (central processingunit, CPU).

The memory 804 may include a volatile memory (volatile memory), forexample, a random access memory (random-access memory, RAM); or includea non-volatile memory (non-volatile memory), for example, a read-onlymemory (read-only memory, ROM), a flash memory, a hard disk drive (harddisk drive, HDD), or a solid stale drive (solid state drive, SSD); orinclude a combination of the foregoing types of memories.

The remote control methods provided in the foregoing embodiments are allperformed by the processor 803. Data such as a video stream, deviceinformation, and operation information is stored in the memory 804. Inaddition, the memory 804 is further configured to store a correspondingprogram instruction or the like to be executed by the processor 803 toperform the remote control method in the foregoing embodiment.

A person of ordinary skill in the art may be aware that, in combinationwith the examples described in the embodiments disclosed in thisspecification, units and algorithm steps may be implemented byelectronic hardware or a combination of computer software and electronichardware. Whether the functions are performed by hardware or softwaredepends on particular applications and design constraint conditions ofthe technical solutions. A person skilled in the art may use differentmethods to implement the described functions for each particularapplication, but it should not be considered that the implementationgoes beyond the scope of the embodiments of this application.

In addition, aspects or features in the embodiments of this applicationmay be implemented as a method, an apparatus or a product that usesstandard programming and/or engineering technologies. The term “product”used in this application covers a computer program that can be accessedfrom any computer readable component, carrier or medium. For example,the computer-readable medium may include but is not limited to: amagnetic storage component (for example, a hard disk, a floppy disk, ora magnetic tape), an optical disc (for example, a compact disc (compactdisc, CD) or a digital versatile disc (digital versatile disc, DVD)), asmart card, or a flash memory (for example, an erasable programmableread-only memory (erasable programmable read-only memory, EPROM), acard, a stick, or a key drive). In addition, various storage mediadescribed in this specification may represent one or more devices and/orother machine-readable media for storing information. The term“machine-readable medium” may include, but is not limited to, a radiochannel and various other media capable of storing, including, and/orcarrying an instruction and/or data.

All or some of the foregoing embodiments may be implemented by usingsoftware, hardware, firmware, or any combination thereof. When softwareis used to implement the embodiments, the embodiments may be implementedcompletely or partially in a form of a computer program product. Thecomputer program product includes one or more computer instructions.When the computer program instructions are loaded and executed on thecomputer, the procedure or functions according to the embodiments ofthis application are all or partially generated. The computer may be ageneral-purpose computer, a dedicated computer, a computer network, orother programmable apparatuses. The computer instructions may be storedin a computer-readable storage medium or may be transmitted from acomputer-readable storage medium to another computer-readable storagemedium. For example, the computer instructions may be transmitted from awebsite, computer, server, or data center to another website, computer,server, or data center in a wired (for example, a coaxial cable, anoptical fiber, or a digital subscriber line (DSL)) or wireless (forexample, infrared, radio, or microwave) manner. The computer-readablestorage medium may be any usable medium accessible by a computer, or adata storage device, such as a server or a data center, integrating oneor more usable media. The usable medium may be a magnetic medium (forexample, a floppy disk, a hard disk, or a magnetic tape), an opticalmedium (for example, a DVD), a semiconductor medium (for example, asolid-state drive (solid state drive, SSD)), or the like.

It should be understood that, in various embodiments of thisapplication, sequence numbers of the foregoing processes do not meanexecution sequences. The execution sequences of the processes should bedetermined based on functions and internal logic of the processes, andshould not constitute any limitation on the implementation processes ofthe embodiments of this application.

It may be clearly understood by a person skilled in the art that, forthe purpose of convenient and brief description, for a detailed workingprocess of the foregoing system, apparatus, and unit, refer to acorresponding process in the foregoing method embodiments, and detailsare not described herein again.

In the several embodiments provided in this application, it should beunderstood that the disclosed system, apparatus, and method may beimplemented in other manners. For example, the described apparatusembodiment is merely an example. For example, the unit division ismerely logical function division and may be other division during actualimplementation. For example, a plurality of units or components may becombined or integrated into another system, or some features may beignored or not performed. In addition, the displayed or discussed mutualcouplings or direct couplings or communication connections may beimplemented through some interfaces. The indirect couplings orcommunication connections between the apparatuses or units may beimplemented in electrical, mechanical, or other forms.

The units described as separate parts may or may not be physicallyseparate, and parts displayed as units may or may not be physical units,in other words, may be located in one position, or may be distributed ona plurality of network units. Some or all of the units may be selectedaccording to actual requirements to achieve the objectives of thesolutions of the embodiments.

When the functions are implemented in a form of a software functionalunit and sold or used as an independent product, the functions may bestored in a computer-readable storage medium. Based on such anunderstanding, the technical solutions of this application essentially,or the part contributing to the prior art, or some of the technicalsolutions may be implemented in a form of a software product. Thecomputer software product is stored in a storage medium, and includesseveral instructions for instructing a computer device (which may be apersonal computer, a server, a network device, or the like) to performall or some of the steps of the methods described in the embodiments ofthis application. The foregoing storage medium includes any medium thatcan store program code, such as a USB flash drive, a removable harddisk, a read-only memory (read-only memory, ROM), a random access memory(random access memory, RAM), a magnetic disk, or an optical disc.

The foregoing descriptions are merely specific implementations of thisapplication, but are not intended to limit the protection scope of thisapplication. Any variation or replacement readily figured out by aperson skilled in the art within the technical scope disclosed in thisapplication shall fall within the protection scope of this application.

1. A method performed by a first terminal, and wherein the methodcomprises: sending, to a server, an image comprising a target device;receiving, from the server, device information corresponding to thetarget device; virtualizing, based on the device information, anoperation interface, wherein the operation interface comprises a controlinterface of a control panel of the target device or a remote control ofthe target device; displaying the operation interface on a current videocall screen; and displaying, on the current video call screen based ondata from a second terminal, one or more operations performed on theoperation interface by a user of the second terminal on the secondterminal.
 2. The method of claim 1, further comprising binding theoperation interface to the target device.
 3. The method of claim 1,wherein displaying the operation interface comprises displaying theoperation interface on or near the target device on the current videocall screen, wherein the operation interface is the control interface ofthe remote control.
 4. The method of claim 1, further comprising:receiving, from the second terminal, and sending, to the target devicebased on the first operation, a control instruction to control thetarget device.
 5. The method of claim 1, further comprising: receiving asecond operation performed by the user on the operation interface;determining a visual effect corresponding to the second operation;displaying the visual effect on the current video call screen; andsynchronizing the visual effect to the second terminal.
 6. The method ofclaim 5, further comprising: determining, based on the second operation,a control instruction corresponding to the second operation; andsending, to the target device, the control instruction to control thetarget device.
 7. An electronic device comprising: a memory configuredto store instructions; and a processor coupled to the memory andconfigured to: send, to a server, an image comprising a target device;receive, from the server, device information corresponding to the targetdevice; virtualize, based on the device information, an operationinterface, wherein the operation interface comprises a control interfaceof a control panel of the target device or a remote control of thetarget device; display the operation interface on a current video callscreen; and display, on the current video call screen based on data froma second electronic device, one or more operations performed on theoperation interface by a user of the second electronic device on thesecond electronic device.
 8. A method, performed by a second terminal ina video call process between the second terminal and a first terminal,and wherein the method comprises: displaying a target device on acurrent video call screen; receiving device information of the targetdevice; virtualizing, based on the device information, an operationinterface, wherein the operation interface comprises a control interfaceof a control panel of the target device or a remote control of thetarget device; displaying the operation interface on the current videocall screen; receiving a first operation performed by a user on theoperation interface on the current video call screen; and displaying avisual effect corresponding to the first operation; and synchronizingthe visual effect to the first terminal.
 9. The method of claim 8,further comprising binding the operation interface to the target device.10. The method of claim 8, wherein displaying the operation interfacecomprises displaying the operation interface on or around the targetdevice on the current video call screen, wherein the operation interfaceis the control interface of the remote control.
 11. The method of claim8, further comprising displaying, on the current video call screen basedon data from the first terminal, one or more second operations performedon the virtual operation interface by a user of the first terminal onthe first terminal. 12.-15. (canceled)
 16. The electronic device ofclaim 7, wherein the processor is further configured to display theoperation interface on or near the target device on the current videocall screen, wherein the operation interface is of the remote control.17. The electronic device of claim 7, wherein the processor is furtherconfigured to overlay the control panel of the target device on thecurrent video call screen with the operation interface, wherein theoperation interface is of the control panel.
 18. The electronic deviceof claim 7, wherein the processor is further configured to replace thecontrol panel of the target device on the current video call screen withthe operation interface, wherein the operation interface is of thecontrol panel.
 19. The electronic device of claim 7, wherein theprocessor is further configured to: receive, from the second electronicdevice, a first operation; and send, to a target device based on thefirst operation, a control instruction to control the target device. 20.The electronic device of claim 7, wherein the processor is furtherconfigured to: receive a second operation performed by the user on theoperation interface; determine a visual effect corresponding to thesecond operation; display the visual effect on the current video callscreen; and synchronize the visual effect to the second electronicdevice.
 21. The electronic device of claim 18, wherein the processor isfurther configured to: determine, based on the second operation, acontrol instruction corresponding to the second operation; and send, tothe target service, the control instruction to control the targetdevice.
 22. The method of claim 1, wherein displaying the operationinterface comprises overlaying the control panel of the target device onthe current video call screen with the operation interface, wherein theoperation interface is the control interface of the control panel of thetarget device.
 23. The method of claim 1, wherein displaying theoperation interface comprises replacing the control panel of the targetdevice on the current video call screen with the operation interface,wherein the operation interface is the control interface of the controlpanel of the target device.
 24. The method of claim 8, whereindisplaying the operation interface comprises overlaying or replacing thecontrol panel of the target device on the current video call screen withthe operation interface, wherein the operation interface is a controlinterface of the control panel.